Dynamic-static unsupervised sequentiality, statistical subunits and lexicon for sign language recognition

a r t i c l e i n f o We introduce a new computational phonetic modeling framework for sign language (SL) recognition. This is based on dynamic–static statistical subunits and provides sequentiality in an unsupervised manner, without prior linguistic information. Subunit " sequentiality " refers to the decomposition of signs into two types of parts, varying and non-varying, that are sequentially stacked across time. Our approach is inspired by the Move-ment–Hold SL linguistic model that refers to such sequences. First, we segment signs into intra-sign primitives, and classify each segment as dynamic or static, i.e., movements and non-movements. These segments are then clustered appropriately to construct a set of dynamic and static subunits. The dynamic/static discrimination allows us employing different visual features for clustering the dynamic or static segments. Sequences of the generated subunits are used as sign pronunciations in a data-driven lexicon. Based on this lexicon and the corresponding segmentation, each subunit is statistically represented and trained on multimodal sign data as a hidden Markov model. In the proposed approach, dynamic/static sequentiality is incorporated in an unsupervised manner. Further, handshape information is integrated in a parallel hidden Markov modeling scheme. The novel sign language modeling scheme is evaluated in recognition experiments on data from three corpora and two sign languages: Boston University American SL which is employed pre-segmented at the sign-level, Greek SL Lemmas, and American SL Large Vocabulary Dictionary, including both signer dependent and unseen signers' testing. Results show consistent improvements when compared with other approaches, demonstrating the importance of dynamic/static structure in sub-sign phonetic modeling. Sign languages are natural languages that manifest themselves via the visual modality in the 3D space. They convey information via visual patterns and serve for communication in parts of Deaf communities [2]. Visual patterns are formed by manual and non-manual cues. The automatic processing of such visual patterns for the Automatic Sign Language Recognition (ASLR) can bridge the communication gap between the deaf and the hearing. Since the early work of [3], there has been progress in visual processing, sign language phonetic modeling, and automatic recognition [1,4,5]. Moreover ASLR may contribute to other disciplines such as linguistics for the study of Sign Languages (SLs), via automated processing of corpora, whereas it is broadly related to human computer interaction. Herein we focus on sign language articulation produced by manual cues. The term " manual cues " refers to the movements and …